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PREFACE 
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unit  monitor  was  Dr  Thomas  R.  Carretta.  Dr  Carretta  is  currently  associated  with  AFRL/HECI  at 
Wright-Patterson  Air  Force  Base,  OH.  He  may  be  reached  via  e-mail  at 
thomas.carretta@he.vyDafb.af.mil. 
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CogScreen-AE  battery,  and  thank  Dr  Andrea  Bemdt  for  her  help  in  specifying  the  LISREL 
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FACTOR  STRUCTURE  OF  THE  COGSCREEN-AE  TEST  BATTERY 


SUMMARY 

CogScreen  Aeronautical  Edition  (CogScreen-AE)  is  a  computerized  test  designed  to 
assess  cognitive  functioning.  The  US  Air  Force  (USAF)  is  evaluating  its  utility  for  establishing  a 
cognitive  baseline  for  pilots  that  would  assist  clinicians  when  evaluating  pilots  with  cognitive 
referral,  which  could  result  in  impaired  flying  ability.  To  better  understand  what  is  measured  by 
the  CogScreen- AE,  confirmatory  factor  analyses  were  performed.  Participants  were  1,015  USAF 
pilot  training  applicants.  Like  many  cognitive  tests,  CogScreen-AE  exhibited  an  hierarchical 
factor  structure,  Somewhat  unusual  was  that  there  was  not  a  strong,  single,  higher  order  factor 
representing  general  cognitive  ability.  Instead,  there  were  two  higher  order  factors  that  measured 
response  efficiency  and  procedural  knowledge.  In  addition  to  the  higher  order  factors,  there  were 
six  first-order  factors  representing  thruput,  response  speed,  shifting  attention,  psychomotor 
tracking,  pathfinder,  and  numeric. 

INTRODUCTION 

Ability  tests  have  long  been  used  in  the  selection  of  pilots,  especially  in  the  military 
(Hilton  &  Dolgin,  1991;  Ree  &  Carretta,  1996).  These  tests  have  been  both  cognitive  and 
psychomotor  and  have  been  administered  via  paper-and-pencil  and  by  computer  (Ree  &  Carretta, 
1998).  Another  use  for  ability  tests  is  to  detect  minimal  brain  dysfunction.  The  US  Federal 
Aviation  Administration  (FAA)  called  for  such  a  test  in  1987.  This  study  investigated  the  factor 
structure  of  a  computerized  test  designed  for  detecting  deficits  in  the  cognitive  functioning  of 
pilots  due  to  minimal  brain  dysfunction. 

CogScreen- Aeromedical  Edition  (CogScreen- AE)  (Kay,  1995)  is  a  computer-based 
cognitive  ability  test  intended  to  assess  changes  or  deficits  in  cognitive,  information  processing, 
and  perceptual  abilities  (e.g.,  attention,  memory,  reaction  time,  simultaneous  information 
processing).  It  was  initially  designed  to  detect  subtle  changes  in  cognitive  functioning  of  aviation 
pilots  that  could  result  in  impaired  flying  ability  (Kay,  1995).  Unlike  other  tests  of  aviation 
aptitude,  CogScreen-AE  does  not  measure  aviation  knowledge  (e.g..  Air  Force  Officer 
Qualifying  Test  or  AFOQT)  (Carretta  &  Ree,  1996)  or  flying  skills  (e.g.,  Canadian  Automated 
Pilot  Selection  System  or  CAPSS)  (Spiimer,  1991).  Rather,  it  focuses  on  the  measurement  of 
cognitive,  information  processing,  and  perceptual  abilities  hypothesized  to  be  related  to  flying 
performance  (Imhoff  &  Levine,  1981).  The  US  Air  Force  (USAF)  is  evaluating  the  utility  of  the 
CogScreen- AE  battery  along  with  a  computerized  version  of  the  Multidimensional  Aptitude 
Battery  (MAB)  (Jackson,  Barton,  &  Blokker,  1992)  for  establishing  a  cognitive  baseline  for 
student  pilots  (Callister,  King,  &  Retzlaff,  1995;  Retzlaff,  Callister,  &  King,  1996). 

Kay  ( 1 995)  conducted  exploratory  factor  analyses  of  28  CogScreen-AE  scores  for  an 
international  sample  of  662  United  States  and  Russian  pilots.  Principal  components  analysis  with 
Varimax  rotation  was  used.  A  9-factor  solution  was  generated  that  accoimted  for  67.1%  of  the 
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v^ance.  The  factors  were:  (1)  visual  scanning  and  sequencing,  (2)  attribute  identification,  (3) 
visual  perceptual  and  spatial  processing,  (4)  motor  coordination,  (5)  choice  visual  reaction  time, 
(6)  visual  associative  memory,  (7)  tracking,  (8)  working  memory,  and  (9)  numerical  operations  ’ 
The  factors  were  not  allowed  to  correlate,  as  an  orthogonal  solution  was  specified.  However, 
interpretation  of  these  factors  is  questionable.  Four  of  the  nine  factors  were  underspecified  (i.e., 
represented  by  two  or  fewer  scores).  Finally,  Kay  did  not  test  the  statistical  goodness-of-fit  of  his 
factor  model  in  his  exploratory  factor  analysis. 

The  purposes  of  this  study  were  to  test  the  goodness-of-fit  of  Kay’s  factor  model  and  to 
establish  and  test  the  goodness-of-fit  of  two  alternate  factor  models  of  the  CogScreen-AE  as 
established  by  cumulative  psychometric  research  on  ability  tests  (Carretta  &  Ree,  1996, 1997; 
Carretta,  Retzlaff,  Callister,  &  King,  1998;  Jensen,  1980;  Ree  &  Carretta,  1994;  Stauffer  Ree  & 
Carretta,  1996;  Vernon,  1969). 


METHOD 


Participants 


Participants  were  1 ,0 1 5  US  Air  Force  pilot  training  applicants.  The  mean  age  at  time  of 
testing  was  20.5  years.  The  sample  was  predominantly  male  (91.9%)  and  White  (88.4%). 
Participants  were  tested  as  a  requirement  for  entrance  into  the  US  Air  Force  Enhanced  Flight 
Screening  Program.  This  program  teaches  basic  flying  skills  and  is  used  to  determine  which 
applicants  should  attend  jet  aircraft  training. 

The  protocol  for  the  current  study  had  been  reviewed  and  approved  by  the  Air  Force 
Human  Use  Committee  of  the  Air  Force  Medical  Operations  Agency.  Informed  consent  was 
obtained  from  participants  before  their  participation. 

Measures 


The  CogScreen-AE  tests  were  administered  on  a  386-based  computer  with  a  14-inch 
color  momtor.  Participants  entered  their  responses  using  a  keypad  and  mouse  or  light  pen. 

Table  1  provides  the  test  developer’s  brief  summary  of  the  attributes  measured  by  the 
CogScreen-AE  tests  and  the  types  of  scores  generated  for  each  test.  Most  of  the  tests  include 
percent  correct,  median  response  time  for  items  answered  correctly,  and  thruput.  Thruput  is 
defined  as  a  measure  of  response  efficiency  and  reflects  the  number  of  correct  responses  per 
minute  (Kay,  1995,  p.8).  Some  CogScreen-AE  tests  (i.e.,  Divided  Attention,  Pathfinder, 
Selective  Attention,  and  Dual  Task)  include  various  process-oriented,  qualitative,  or  response 
error  measures  that  cannot  be  classified  as  accuracy,  speed,  or  thruput.  Brief  descriptions  of  the 
tests  follow.  Kay  (1995)  provided  detailed  test  descriptions  and  scoring  procedures. 

Backward  digit  span.  This  test  is  similar  to  the  backward  digit  span  task  found  in  the 
Wechsler  Adult  Intelligence  Scale-Revised  (WAIS-R)  (Wechsler,  1981).  In  this  test,  participants 
are  sequentially  presented  with  a  series  of  3  to  6  digits.  Their  task  is  to  reproduce  the  sequence  in 
reverse  order.  The  score  is  the  percent  of  series  recalled  in  correct  reverse  order  (BDSACC). 
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Math.  This  test  consists  of  traditional  multistep  word  problems.  Scores  include  percent 
correct  (MATHACC),  median  response  time  on  items  answered  correctly  (MATHRTC),  and 
thruput  (i.e.,  MATHPUT,  number  of  problems  answered  correctly  per  minute). 


Table  1.  Brief  Descriptions  of  CogScreen-AE  Tests 


Test 

Abbreviatior 

1  Attributes  Measured 

Types  of  Scores 

Backward  Digit  Span 

BDS 

verbal  sequential  processing, 
visual  attention,  &  working  memory 

response  accuracy 

Math 

MATH 

math  skills,  reading  comprehension, 
reasoning,  &  working  memory 

response  accuracy,  response 
speed,  &  thruput 

Visual  Sequence 
Comparison 

VSC 

verbal  sequential  processing,  visual 
attention,  visual  perceptual  speed, 

&  working  memory 

response  accuracy,  response 
speed,  &  thruput 

Symbol  Digit  Coding 

SDC 

attention,  immediate  and  delayed 
memory,  information  processing 

response  accuracy,  response 
speed,  &  thruput 

Matching-to-Sample 

MTS 

spatial  processing,  visual-perceptual 
speed,  &  working  memory 

response  accuracy,  response 
speed,  &  thruput 

Manikin 

MAN 

spatial  orientation  &  visual-spatial 
perception 

response  accuracy,  response 
speed,  8l  thruput 

Divided  Attention 

DAT 

choice  reaction  time,  divided  attention, 
multitasking,  &  working  memory 

response  accuracy,  response 
speed,  &  thruput 

Auditory  Sequence 
Comparison 

ASC 

auditory  attention,  sound  pattern 
discrimination,  &  working  memory 

response  accuracy,  response 
speed,  &  thruput 

Pathfinder 

PF 

ability  to  shift  mental  set,  memory, 
motor  coordination,  organizing 
and  sequencing,  &  visual  scanning 

response  accuracy,  response 
speed,  &  thruput 

Shifting  Attention 

SAT 

application  of  rules,  choice  reaction  time, 
concept  formation,  deductive  reasoning, 
&  working  memory 

response  accuracy,  response 
speed,  &  thruput 

Dual  Task 

DTT 

divided  attention,  multitasking,  & 
working  memory 

response  accuracy,  response 
speed,  thruput,  &  tracking  error 

Visual  sequence  comparison.  Two  alphanumeric  strings  are  simultaneously  presented 
side-by-side  on  the  screen.  Participants  must  determine  whether  the  strings  are  the  same  or 
different.  String  length  varies  from  4  to  8  characters.  Performance  measures  include  percent 
correct  (VSCACC),  response  time  for  correctly  answered  items  (VSCRTC),  and  thruput 
(VSCPUT). 
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Symbol  digit  coding.  This  test  consists  of  three  tasks:  (1)  symbol-digit  coding,  (2) 
immediate  recall,  and  (3)  delayed  recall.  The  symbol-digit  coding  task  is  a  computer-based 
analogue  of  Digit  Symbol  test  from  the  WAIS-R  (Wechsler,  1981).  In  this  task,  six  symbol-digit 
pmrs  are  shown  at  the  top  of  the  screen.  The  pairs  remain  on  the  screen  throughout  the  symbol- 
digit-coding  task.  Participants  are  instructed  to  remember  the  symbol-digit  pairs  for  a  later 
memory  test  (immediate  and  delayed  recall  tasks;  see  below).  Meanwhile,  in  the  center  of  the 
screen,  a  row  of  symbols  is  shown  with  blank  spaces  below  each  symbol.  Participants  enter  the 
digit  that  corresponds  to  each  symbol.  Scores  include  accuracy  (SDCACC),  and  the  number  of 
items  completed  correctly  per  minute  (i.e.,  thruput,  SDCPUT). 

The  immediate  recall  task  measures  the  ability  to  recall  the  symbol-digit  pairs  presented 
in  the  first  part  of  the  test.  The  six  symbols  used  in  the  symbol-digit-coding  task  are  presented. 
Participants  are  required  to  remember  the  corresponding  digit.  Response  accuracy  (SDCIRACC) 
is  the  only  score.  ^ 


The  delayed  recall  task  is  identical  to  the  immediate  recall  task,  except  that  it  is  presented 

after  the  Pathfinder  test.  As  with  the  immediate  recall  task,  the  only  score  is  response  accuracy 
(SDCDRACC). 


Matching-to-sample.  Participants  are  shown  a  4  x  4  grid  of  filled  and  empty  cells  (i.e., 
target  grid).  After  a  brief  presentation,  the  target  grid  is  removed  and  replaced  by  two  grids  that 
are  displayed  side-by-side  on  the  screen.  One  pattern  is  identical  to  the  target  grid  while  the  other 
differs  in  1  of  the  16  cells.  Participants  must  choose  the  grid  matching  the  target  grid.  Scores 
include  accuracy  (MTSACC),  response  time  (MTSRTC),  and  thruput  (MTSPUT). 


Manikin.  This  test  is  based  on  a  spatial  transformation  task  described  by  Benson  and 
Gedye  (1963).  Participants  are  presented  with  a  male  figure  in  one  of  four  positions.  The  image 
can  be  either  right  side  up  or  upside  down  and  facing  toward  or  facing  away.  The  figure  holds  a 
flag  in  one  h^d.  Participants  must  determine  which  hand  (right  or  left)  is  holding  the  flag.  Test 
performance  is  scored  for  accuracy  (MANACC),  speed  (MANRTC),  and  thruput  (MANPUT). 


attention  test.  TOs  test  has  two  parts:  (1)  a  visual  indicator  monitoring  task  and 
(2)  the  visual  indicator  monitoring  task  presented  simultaneously  vrith  the  Visual  Sequence 
Comparison  task  (see  above).  In  the  first  part  (visual  monitoring  alone  task),  participants  watch  a 
cursor  move  vertically  within  a  circle  divided  into  upper,  central,  and  lower  sections.  When  the 
cursor  moves  from  the  cenfral  section  into  either  the  upper  or  lower  section,  participants  touch  a 
box  labeled  CENTER”  with  a  light  i^n  to  recenter  the  cursor.  Scores  include  the  median 

m  A  spends  in  either  the  upper  or  lower  sections  before  being  recentered 

(UA 1 IRTC)  and  the  number  of  premature  (i.e.,  anticipatory)  responses  (DATIPRE). 


In  part  2  (dual-task),  the  visual  indicator-monitoring  task  is  paired  with  the  Visual 
Sequence  Comparison  task.  Performance  measures  include  response  speed  (DATDRTC)  and 
number  of  premature  responses  (DATDPRE)  for  the  indicator  monitoring  task  and  response 
accuracy  (DATSCACC),  response  speed  (DATSCRTC),  and  thruput  (DATSCPUT)  for  the 
visual  sequence  comparison  task. 
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Auditory  sequence  comparison.  This  test  involves  the  comparison  of  two  series  of  4  to  8 
tones.  The  series  are  presented  sequentially  and  may  be  identical  or  differ  in  the  pitch  of  one 
tone.  Participants  make  a  “SAME”  or  “DIFFERENT’  judgment  for  each  pair.  Scores  include 
accuracy  (ASCACC),  response  speed  (ASCRTC),  and  thruput  (ASCPUT). 

Pathfinder.  Participants  are  taught  three  sequencing  rules  regarding  numbers,  letters,  and 
a  combination  of  numbers  and  letters.  For  each  item,  a  number  or  letter  is  shown  in  the  center  of 
the  screen.  Participants  choose  from  among  four  characters  (numbers  and/or  letters),  the  one  that 
should  appear  next  in  the  sequence.  Three  of  the  four  character  choices  are  updated  following 
each  response. 

There  are  separate  accuracy,  speed,  thruput,  and  coordination  scores  for  letter,  number, 
and  “combination”  items.  The  response  accuracy  scores  are  PFLACC  (letters),  PFNACC 
(numbers),  and  PFCACC  (combined).  The  median  response  time  scores  are  PFLMRT  (letters), 
PFNMRT  (numbers),  and  PFCMRT  (combined).  The  thruput  scores  are  PFLPUT  (letters), 
PFNPUT  (numbers),  and  PFCPUT  (combined).  Coordination  scores  measure  participants 
proximity  to  the  center  of  the  target  numbers  and  letters  and  include  PFLCOOR  (letters), 
PFNCOOR  (numbers),  and  PFCCOOR  (combined). 

Shifting  attention  test.  In  this  test,  participants  must  alter  their  responses  depending  on 
rule  changes.  Under  Rule  1 ,  boxes  are  selected  based  on  the  color  of  their  borders.  Under  Rule  2, 
boxes  are  selected  based  on  the  direction  of  their  arrow,  and  under  Rule  3,  they  are  selected 
based  on  the  color  of  their  arrow.  The  test  items  begin  after  the  three  rules  have  been  presented 
and  a  practice  session  completed.  During  the  first  group  of  test  items,  one  of  the  rules  is  shown 
before  each  item.  In  the  second  group  of  test  items  (discovery),  participants  must  discover,  and 
then  apply,  the  active  response  rule,  which  changes  after  a  variable  number  of  correct  answers. 
During  the  discovery  items,  participants  must  use  trial-and-error  to  determine  the  active  rule. 

Response  accuracy,  speed,  and  thruput  are  measured  for  all  tasks  in  this  test.  Additional 
scores  for  the  discovery  items  include  number  of  shifts  completed  (SATDIRUL),  failures  to 
maintain  set  (SATDIFAI),  number  of  nonconceptual  responses  (SATDINON),  and  number  of 
perseverative  errors  (SATDIPER). 

Dual  task.  This  test  is  similar  to  various  measures  of  time-sharing  ability  (see,  for 
example,  Carretta  &  Ree,  1993;  North  &  Gopher,  1976).  In  this  test,  participants  must  learn  to 
separately  perform  a  tracking  task  and  a  memory  task,  then  perform  both  simultaneously.  In  the 
first  part  (compensatory  tracking  alone),  participants  use  the  left  and  right  arrow  keys  to  keep  a 
cursor  in  the  center  of  the  screen.  Without  intervention,  the  cursor  will  drift  off  the  screen 
causing  a  “boundary  hit”  Scores  for  part  1  include  mean  absolute  tracking  error  (DTTAABS) 
and  the  number  of  boundary  hits  (DTTAHIT). 

The  second  part  involves  a  delayed  recall  task.  In  this  task,  participants  are  shown  the 
numbers  1, 2,  or  3.  The  first  number  is  then  replaced  by  a  1, 2,  or  3.  Participants  must  recall  the 
previous  number  shown  and  select  it  (using  a  light  pen),  while  simultaneously  mentally  encoding 
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the  current  number  for  the  next  item  presentation.  Performance  measures  include  accuracy 
(DTTPAACC),  response  speed  (DTTPARTC),  and  thruput  (DTTPAPUT).  The  third  part 
simultaneously  presents  the  tracking  and  delayed  recall  tasks.  Scores  include  tracking  error 
(DTTDABS),  boundary  hits  (DTTDHIT),  recall  accuracy  (DTTPDACC),  response  speed 
(DTTPDRTC),  and  thruput  (DTTPDPUT). 

Procedures 

Participants  completed  the  CogScreen-AE  tests  shortly  before  beginning  the  USAF 
Enhanced  Flight  Screening  Program.  CogScreen-AE  testing  was  done  to  establish  an  ideographic 
cognitive  baseline  for  the  clinical  evaluation  of  pilots  for  comparative  purposes  after  sustaining  a ' 
head  injury  or  other  neurological  insult  (Retzlaff,  et  al.,  1996). 

Analyses  included  descriptive  statistics,  correlations,  and  confirmatory  factor  analyses. 
Because  the  participants  had  been  selected,  at  least  in  part,  on  the  basis  of  aptitude  test  scores, 
they  constituted  a  range-restricted  sample.  Such  samples  provide  relatively  poor  statistical 
estimates  of  the  relations  between  variables  (Thorndike,  1949).  The  multivariate  correction 
method  (Lawley,  1943;  Ree,  Carretta,  Earles,  &  Albert,  1994)  was  used  to  correct  the  means, 
standard  deviations,  and  correlations  for  the  effects  of  prior  selection  to  a  US  Air  Force  applicant 

sample  (Skinner  &  Ree,  1987).  The  corrected  correlation  matrix  was  used  in  the  confirmatory 
factor  analyses. 

Kay  (1995)  lists  65  summary  scores  from  the  CogScreen-AE  tests  (19  accuracy,  19 
speed,  16  thruput,  and  1 1  process-oriented).  As  noted  earlier,  the  thruput  scores  are  linear 
transformations  of  the  accuracy  and  speed  scores.  Therefore,  it  would  be  inappropriate  to  include 
accuracy  and  speed  scores  in  confirmatory  factor  analyses  with  thruput  scores  based  on  the  same 
test  Items.  We  chose  to  follow  Kay’s  lead  by  focusing  on  the  same  28  scores  as  used  in  his 
previous  exploratory  factor  analysis  (3  accuracy,  5  speed,  12  thruput,  and  8  process  scores). 
Several  models  were  specified,  parameters  estimated,  and  the  goodness-of-fit  to  the  data  was 
tested.  Hierarchical  confirmatory  factor  analyses  were  performed  using  LISREL  8  (Joreskoe  & 
Sorbom,  1996). 

Model  1  consisted  of  the  nine  first-order  orthogonal  factors  described  by  Kay  (1995). 
Model  2  was  identical  to  Model  1  except  that  the  first-order  factors  were  allowed  to  correlate  as 
IS  usually  found  m  cogmtive  ability  tests.  Model  3  was  a  simple,  more  parsimonious  model  with 
SIX  first-order  factors.  Some  of  these  factors  were  developed  by  grouping  similar  types  of  scores 
across  tests  (i.e.,  thruput,  response  time,  tracking,  numerical),  while  others  represented  specific 
CogScreen-AE  tests  with  multiple  scores  per  test  (i.e..  Shifting  Attention  has  7  scores.  Pathfinder 
has  6  scores,  etc.).  The  first-order  confirmatory  factor  analyses  (CFAs)  allowed  all  observed 
y^ables  (28  CogScreen-AE  scores)  to  load  on  their  first-order  factors.  This  retains  the  factor 
influence  on  all  scores  and  was  necessary  to  conduct  the  latter  hierarchical  CFAs  based  on  the 
correlations  of  the  six  first-order  factors. 

Two  goodness-of-fit  indices  were  computed  to  test  the  goodness  of  the  models  These 
were  the  Comparative  Fit  Index  (CFI)  (Bentler,  1990)  and  Root  Mean  Square  Error  of 
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Approximation  (RMSEA)  (Browne  &  Cudeck,  1993).  Values  above  .90  for  the  CFI  and  below 
,08  for  the  RMSEA  are  considered  indicators  Of  acceptable  fit. 

RESULTS  AND  DISCUSSION 

The  sample  data  were  corrected  for  range  restriction.  The  means  and  standard  deviations 
are  presented  in  Table  2  and  the  correlation  matrix  is  presented  in  Table  3.  As  would  be  expected 
in  selected  samples,  after  correction  for  range  restriction,  the  means  decreased  and  the  standard 
deviations  increased.  In  general,  these  changes  were  small.  After  correction  for  range  restriction, 
the  correlations  increased,  but  again  by  only  a  small  amoimt.  Because  the  Air  Force  participants 
were  selected  on  highly  g-loaded  tests,  this  suggests  that  the  CogScreen-AE  is  not  heavily  g- 
loaded.  If  it  were,  the  restriction  caused  by  incidental  selection  would  have  been  greater 
(Thorndike,  1949)  leading  to  larger  differences  between  pre-  and  post-correction  values.  All 
confirmatory  factor  models  were  estimated  using  the  corrected  data. 

Table  2.  Means  and  Standard  Deviations 


Score 

Abbr. 

Mean 

SD 

Mean 

SD 

Backward  Digit  Snan 

1.  Accuracy 

BDSACC 

0.85 

0.17 

0.81 

0.18 

Math 

2.  Thruput 

MATHPUT 

2.19 

0.98 

1.86 

1.09 

Visual  Seouence  Comoarison 

3.  Thruput  VCSPUT 

29.84 

6.65 

29.10 

6.77 

Svmbol  Digit  Coding 

4.  Thruput 

SDCPUT 

76.63 

23.62 

72.04 

24.17 

5.  Immediate  Recall  Acc. 

SDCIRACC 

0.12 

0.31 

0.11 

0.31 

Matching-to-Samole 

6.  Thruput 

MTSPUT 

49.47 

10.26 

47.53 

10.29 

Manikin 

7.  Thruput 

MANPUT 

34.27 

8.94 

30.97 

9.19 

Divided  Attention  Test 

8.  Sequence  Comparison 

DATSCPUT 

27.43 

7.01 

26.69 

7.11 

Thruput 

9.  Indicator  “Alone”  RT 

DATIRTC 

0.37 

0.09 

0.37 

0.09 

10.  Indicator  “Dual”  RT 

DATDRTC 

0.63 

0.19 

0.63 

0.20 
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Table  2.  Means  and  Standard  Deviations  (Cont’d) 


Score 


Abbr. 


Mean 


SD 


Mean 


SD 


Auditory  Sequence  Gomp. 

11.  Thruput 

Pathfinder 

12.  Letter  RT 

13.  Number  RT 

14.  Combined  RT 

15.  Letter  Coordination 

16.  Number  Coordination 

17.  Combined  Coordination 

Shifting  Attention  Test 


ASCPUT 


PFLRTC 

PFNRTC 

PFCRTC 

PFLCOOR 

PFNCOOR 

PFCCOOR 


92.90 

27.10 

90.25 

27.25 

0.70 

0.18 

0.71 

0.18 

0.76 

0.23 

0.77 

0.23 

1.03 

0.30 

1.10 

0.31 

3.44 

7.86 

3.47 

7.86 

3.50 

8.55 

3.45 

8.55 

3.36 

7.81 

3.44 

7.81 

105.70 

21.95 

103.60 

21.95 

6.85 

2.67 

5.92 

2.80 

4.71 

16.28 

2.92 

16.33 

2.12 

1.91 

2.75 

1.98 

1.76 

2.22 

1.71 

2.22 

83.56 

17.43 

79.93 

17.52 

98.26 

17.56 

95.99 

17.58 

15.25 

13.68 

17.82 

14.00 

45.16 

26.82 

52.22 

27.56 

153.23 

186.40 

152.37 

186.70 

121.49 

157.44 

102.53 

157.96 

1 9.  Discovery  Rule  Shifts  S ATDIRUL 

20.  Discovery  Rule  Accuracy  SATDIACC 

2 1 .  Discovery  Failures  S ATDIFAI 

22.  Discovery  Perseverative  SATDIPER 
Errors 

23.  Instruction  Thruput  SATINPUT 

24.  Arrow  Color  Thruput  SATACPUT 

Dual  Task 

25.  Tracking  “Alone”  Error  DTTAABS 

26.  Tracking  “Dual”  Error  DTTDABS 

27.  Previous  Number  DTTPAPUT 
“Alone”  Thruput 

28.  Previous  Number  DTTPDPUT 

“Dual”  Thruput _ _ _ 

^.^Means  and  standard  deviations  were  corrected  for  range  restriction  using  the  multivariate  method  (Lawley, 


nonr  f  ^ ^odel  1  (Kay’s  [1995]  9.factor  orthogonal  model)  indicated  a 

^  ^SEA  was  .207.  This  was  not  a  surprising  result  for  an 

orAogonal  (unco^elated)  model  as  there  is  a  long  history  of  correlated  abilities  stretching  back 
to  the  founding  of  factor  analysis  (Spearman,  1 904).  ^ 


Model  2  an  oblique  or  correlated  factors  version  of  Model  I,  fared  considerably  better. 
Tbe  CFI  was  .958  and  the  RMSEA  was  .049,  both  indicative  of  an  acceptable  fit.  However,  one 
of  the  correlation  among  the  factors  exceeded  1,0,  probably  a  consequence  of  model  mis- 
^ifiMUon.  It  IS  apparmt  that  this  model  does  not  fit  the  data  thus  rendering  the  interpretation 

of  the  fit  indexes  m^t.  Kay’s  (1995)  9.factor  model,  whether  orthogonal  or  oblique,  Ls  not 
oner  a  good  fit  to  the  data. 
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Table  3.  Correlation  Matrix 


ON  Tf  (N 

o  m 
o  o 

m  <N 
O  ^ 

o  o  ^ 


^  s 
'T  T  9 

^  o 


^  VO  On 
VO 


VO  —  Tf 
r-  Tf 


^  fT)  ^  VO  Ov 
00  00  m  o 
o  o  o  o  ^ 

'O  On  Os  t;^  O 

Os  o  ^ 

O  <N  O  O  ^ 


V^OOfSOs  — 

C>  o  <n 

o  o  O  O  CN 

I  I 

tN  o  <N  Os 

^  Os  O  O  ^ 

O  o  O  o  CN 

Os  ^  VO  Os  f<S 

^  o  00  rs  ^ 

•7000- 

Os  t-  CN  Os 
00  O  ^  VO 

O  O  O  O  ^ 


■  CN  Os  2 

►  52  ro  g 

►  o  VO  2 

I  Tf  o  r- 
I  o  o  m 

►  o  O  VO 

'  ^  CN 

•  9  o  g 

'  2  ®  9 

cn  CN  00 

Os 

VO  o  o 


1 

Os 

0 

1 

00 

Os 

0 

1 

CO 

0 

1 

VO 

00 

CO 

tT 

9 

t 

VO 

VO 

1 

0 

CO 

1 

CN 

r-- 

CN 

VO 

VO 

VO 

0 

VO 

CO 

1 

VO 

CO 

CO 

VO 

VO 

»o 

VO 

VO 

N- 

CN 

2 

00 

00 

1 

-173 

-187 

Tf 

VO 

Os 

CO 

0 

1 

0 

VO 

0 

VO 

0 

r- 

TT 

r- 

0 

0 

0 

Os 

00 

0 

CO 

00 

0 

1 

VO 

1 

r- 

Os 

VO 

r- 

0 

CN 

VO 

-V!f 

N- 

0 

VO 

VO 

Tf 

CO 

Os 

*0 

VO 

s 

tT 

0 

CO 

VO 

VO 

VO 

0 

r- 

CO 

Os 

0 

Tf 

Os 

VO 

CN 

VO 

T 

Os 

0 

VO 

1 

CO 

1 

VO 

Os 

Cjl 

00 

0 

CO 

1 

CO 

0 

VO 

0 

1 

r-* 

Os 

0 

Os 

0 

VO 

CO 

0 

1 

0 

0 

0 

VO 

r- 

r- 

00 

CO 

CO 

CN 

VO 

VO 

CN 

Os 

9 

VO 

0 

CO 

CO 

0 

t-' 

CN 

0 

?3 

0 

VO 

^ 

CO 

00 

0 

9 

1 

0 

0 

r- 

0 

00 

CO 

0 

CN 

0 

0 

s 

0 

1 

00 

VO 

0 

VO 

CN 

0 

CN 

0 

CN 

0 

VO 

CN 

0 

1 

Os 

0 

0 

Os 

0 

0 

r- 

CN 

0 

1 

0 

0 

0 

CO 

CO 

0 

1 

Tf 

0 

1 

00 

CO 

0 

s 

0 

0 

0 

Os 

0 

CN 

Os 

0 

VO 

Os 

0 

0 

CN 

0 

’Vt 

CO 

0 

VO 

0 

0 

CO 

0 

0 

9 

1 

0 

VO 

0 

0 

VO 

9 

TT 

VO 

0 

r- 

VO 

0 

CO 

CO 

0 

VO 

0 

r- 

0 

VO 

VO 

0 

CO 

CN 

0 

CN 

0 

00 

1 

00 

CO 

0 

1 

0 

0 

0 

Os 

s 

CN 

0 

1 

TT 

0 

VO 

0 

CO 

CO 

0 

0 

00 

CO 

0 

1 

t 

2 

0 

CN 

9 

00 

0 

CO 

VO 

0 

OS 

0 

VO 

CO 

0 

Os 

0 

0 

00 

0 

0 

1 

CN 

CO 

VO 

0 

Os 

CN 

0 

CJs 

0 

VO 

0 

1 

VO 

VO 

0 

1 

00 

0 

1 

0 

00 

0 

1 

0 

00 

0 

1 

Os 

0 

Os 

*0 

0 

0 

0 

0 

CO 

VO 

0 

1 

0 

0 

r- 

Os 

0 

0 

0 

1 

VO 

00 

0 

0 

0 

1 

1 

CN 

Os 

0 

r-* 

0 

r- 

Os 

0 

VO 

CO 

0 

0 

0 

1 

CN 

0 

CN 

CO 

0 

00 

CN 

0 

1 

CO 

CN 

0 

VO 

CO 

0 

1 

VO 

CN 

0 

Os 

CN 

0 

1 

r-' 

CO 

0 

r- 

0 

00 

0 

1 

0 

1 

VO 

0 

1 

0 

0 

0 

VO 

r- 

0 

r- 

00 

1 

CN 

0 

CN 

0 

VO 

0 

CO 

CO 

Os 

VO 

1 

00 

0 

0 

CN 

VO 

0 

Os 

0 

0 

0 

Os 

0 

CN 

00 

CO 

CO 

N- 

Os 

CO 

t- 

0 

VO 

r- 

CO 

VO 

CO 

CN 

VO 

VO 

VO 

1 

00 

VO 

CO 

Tt 

00 

CO 

1 

r- 

N* 

»o 

VO 

VO 

0 

00 

CN 

1 

00 

VO 

CN 

1 

0 

00 

CN 

1 

0 

0 

0 

CO 

0 

0 

CN 

0 

N- 

CN 

0 

1 

Os 

VO 

VO 

ro 

CN 

0 

00 

0 

t- 

VO 

xt 

vn 

0 

0 

Os 

1 

VO 

00 

1 

CO 

0 

VO 

1 

CN 

00 

CO 

00 

CN 

1 

00 

Os 

1 

CN 

CO 

1 

1 

Os 

0 

1 

Os 

CO 

00 

N* 

CO 

1 

1 

VO 

OS 

CO 

Os 

OS 

VO 

CN 

VO 

Os 

Os 

CO 

Os 

0 

0 

0 

06 

r- 

CN 

1 

Os 

00 

0 

N- 

00 

0 

1 

00 

0 

CO 

CN 

0 

CO 

0 

CO 

1 

VO 

00 

1 

VO 

CN 

VO 

0 

Os 

CO 

0 

1 

Os 

VO 

0 

1 

CN 

0 

-175 

VO 

r- 

1 

? 

VO 

1 

Os 

VO 

r- 

CO 

00 

CN 

1 

OS 

1 

-123 

CO 

VO 

0 

00 

CN 

VO 

CN 

CO 

1 

CN 

rs 

00 

00 

CN 

0 

VO 

CN 

N- 

CO 

Os 

0 

0 

0 

CO 

OS 

VO 

VO 

CN 

1 

CN 

Os 

0 

00 

0 

1 

CN 

00 

0 

0 

0 

Os 

CN 

1 

-171 

VO 

CN 

CN 

CO 

VO 

0 

r^ 

CN 

0 

1 

00 

VO 

0 

1 

VO 

VO 

0 

00 

00 

1 

VO 

00 

1 

CN 

0 

VO 

1 

00 

»o 

00 

Os 

Os 

-133 

0 

00 

0 

00 

CO 

VO 

N- 

CO 

1 

CO 

Os 

00 

00 

0 

VO 

CN 

0 

0 

0 

CO 

CO 

Os 

VO 

Os 

C?s 

00 

CN 

00 

00 

0 

VO 

00 

0 

1 

00 

0 

CN 

CN 

0 

VO 

0 

CO 

1 

«o 

00 

1 

ro 

VO 

CN 

r- 

VO 

0 

? 

0 

1 

0 

1 

00 

CO 

1 

«? 

OS 

VO 

CO 

1 

VO 

CN 

2 

0 

0 

CO 

CN 

00 

CO 

1 

CO 

r^ 

CN 

1 

CN 

CN 

1 

2 

CO 

CN 

r- 

CN 

Os 

CN 

00 

CN 

»o 

CO 

VO 

N* 

0 

0 

0 

0 

VO 

CN 

CO 

N- 

CN 

VO 

CN 

Os 

CO 

N* 

CN 

r- 

0 

1 

00 

0 

CN 

VO 

0 

N- 

VO 

1 

VO 

VO 

1 

0 

Os 

CN 

VO 

CN 

t 

00 

2 

1 

-059 

0 

1 

00 

0 

CO 

1 

CN 

CO 

■ 

00 

CN 

CN 

VO 

VO 

CO 

1 

VO 

V>m4 

CN 

1 

-211 

VO 

CO 

VO 

CN 

VO 

CN 

CN 

Tf 

00 

r- 

0 

0 

0 

C^ 

VO 

00 

VO 

00 

CN 

CN 

Os 

»o 

»o 

N* 

059- 

VO 

VO 

0 

1 

CN 

00 

0 

N- 

0 

0 

1 

>- 

VO 

1 

r- 

VO 

.  1 

0 

0 

0 

-127 

-141 

c^o^vnoN- 
Os  cn  cn 
o  O  M  rn  CN 

^  CN  »n  CN 
52  CN  ro  ^ 

O  ^  ro  Tf 

y  ^  t-  r-  CN 
o  o  ^  r-. 

O  «  CN  - 

CN  vn  o  r- 

^  Tt  00  2 

®  O  CN  CN  _ 
ro  ^  ^  ^  Os 

:r  r- 

O  m  CN  -7 
O  CN  ^  9 

^  ^  ^  OS 

^  CN  00  3  3 

o  ^  ^  3 

CN  C^  2  O 

VO  CN  o  r;  o 

o  CN  CN  2  2 

Tf  ^  ^  o  2 

N*  Os  r>-  o  SS 

O  CN  fO  o  9 

00  00  O  Os  JN 

CN  vn  o  00 

o  ^  o  cn  ^ 

o  vn  00 

VO  O  *"«  Os 
o  CN  m  ^ 


o  00  rf  -r 
O  N*  ^ 
CN  CN  N* 

O  --  CN  S 

CN  Os  rr  00 

^  CN  CN  ^ 

o  10  OS 

N*  o  r-  S 

CN  CN  CN  5 

CN  VO  m  O 
Os  TT  CN  o 
fO  CN  CN  o 

I  I  i  ^ 

— *  CN  O  00 
CN  ^  O  ^ 
cn  CN  O  CN 

^  I 

VO  O  CN  2 

m  o  m  52 

m  O  CN 


Q  wn  m 

S  00 


^  CN  00 
CO  Os 
CN  CN  CN 


OOcnvOr^t^^^OVON- 
Os*-^OsN’Tfosr^S'^*'^'^ 
^CN*-*vnooO®vOVOO 

*  I  I  »  * 

N-CN^OOvo5Sg52oc<^CN 

f^rOMm5g90^f,,<N 

g'vooor^!PSSS'®?!So' 

c^fncNcnlL?S^S^^22o 


VO  Tf 
Tt  t^  VO 

o  ^  ^ 

t  I 

CO  Tf  o 
o  CN  00 


CO  vr>  00 

Os  m 

CN  O  O 

o  —  CO 
o  CO  CN 
CO  « 

I  t  I 

O  Os  O 
pi  Os  00 


^  Tf 
O 

O  O  'O 

I  I  I 
2  Tj*  OS 
O  o  CO 

S  2 

9  ^ 


rg  00  r- 

S  ^  Ov 

S  9  9 

Tf 

522:9 

5S  Os  CO 
H  00  vr» 
2  o  o 

o  o  OS 
CN  CO 


OVnc000]^CNOV00s  00  00  00OCN^CNcNV0^^'^0sCN 

ococococNTr9cococNfNCNvnoOOovnv->’7^^CN 

•-^OOCNCNVOvoOsCNt^Ot^^V^VO^V^Osr^OO^CO'OcN 

fOCN*7^*^'^CNCNCNoor^OOCOOOoocoCNcNOOO 


CN  ^  CO  CN 

N-  00  Os  CO  00 

CO  CN  CN  CN 

^  CO  CO  ^  ^ 

Os  Os  00  CN 

Tf  CN  »0  CN  CO 

I  I 

«— I  v/-^  ^ 

CN  Os  CN 

—  CN  — *  O  ^ 


Os  NT  c^  CN  vn  CO 

VO  CO  VO  Os  VO  Os 

rf  CO  CO  VO  VO  VO 

I  I  I  I  l.  I 

^  00  O  CO  ^ 
^  ^  CN  Os  00  r-  00 

g  CO  CO  CO  ^  ^ 


^  o  o  CN  *--*  VO 

5  CN  2  p;j  00  r-  00 


>  }n  VO  Os 

I  Ch  CO  ^ 

900 

o  9  «? 

<r>  O 

00  J5  SI* 
099 

O  fN 
^  Os  ^ 
®  9  o 


00  VO  r' 

O  00  Os  VO  ^  VO 

VO  CO  CN  *7  ^ 

CO  CO  Os  ^  C^ 

«0  O  CN  c^  CO 

Tf  i-3  o  ^  ^ 

I  ( 

55  r-  V£>  ^  ^  CO 

2  15  <=>  o 

CN  —  CN  -7  O  ® 


oP?SSSi^'^®®£j^co;3;co^r-covoosvo^osoo<NvooooscNcN 

§?5§SSo25552SS2SSSS2S252g  =  2gg 

•V  lit  tit  ^llt^ 


l^fNCO^vovor^OOOs-^  —  -^^-^^--I^^^CNCNCNCNCNCNCNCNCN 


9 


Nojes.  Names  associated  with  scores  1-28  can  be  found  in  Table  2.  Correlations  above  the  diagonal  are  observed,  while  those  below  the  diagonal 
corrected  for  range  restriction  (Lawley,  1943).  Decimal  points  were  omitted. 


Model  3  with  six  first-order  factors  had  a  CFI  of  .908  and  a  RMSEA  of  .072,  again 
indicative  of  an  acceptable  fit.  Models  2  and  3  cannot  be  tested  directly  against  each  other  as  one 
is  not  a  proper  subset  of  the  other.  Therefore,  it  is  inappropriate  to  compare  fit  indices  between 
these  models.  All  correlations  among  the  six  first-order  factors  in  Model  3  were  within  the 
bounds  of  ±1.0.  We  prefer  this  model  for  its  interpretability  and  its  parsimony.  Table  4  shows  the 
loadings  of  the  scores  on  the  factors  and  Table  5  shows  the  correlations  among  the  first-order 

Table  4.  Factor  Loadings  for  6-Factor  First-Order  Model 


Factor 


Score 


I  II  III  IV  V  VI 


1.  BDSACC 

2.  MATHPUT 

3.  VSCPUT 

4.  SCDPUT 

5.  SDCIRACC 

6.  MTSPUT 

7.  MANPUT 

8.  DATSCPUT 

9.  DATIRTC 

10.  DATDRTC 

11.  ASCPUT 

12.  PFLRTC 

13.  PFNRTC 

14.  PFCRTC 

15. PFLCOOR 

16. PFNCOOR 

17.  PFCCOOR 

18.  SATADPUT 

19.  SATDIRUL 

20.  SATDIACC 

21.  SATDIFAI 

22.  SATDIPER 

23.  SATINPUT 

24.  SATACPUT 

25.  DTTAABS 

26.  DTTDABS 

27.  DTTPAPUT 

28.  DTTPDPUT 


.545 

.349 

.652 

.488 

.38-1 

.682 

.493 

.483 

.688 

.570 

.572 


.828  -.142 
.961 
.075 
-.845 
.015 
-.121 
-.155 


.244 

.283 


-.043 

.226 

.575 

-.959 


.085 

.103 

.162 

.998 

.935 

.998 


1.188 

.550 


Note,  Factors  I  through  VI  are  thruput,  response  time,  shifting  attention,  tracking,  pathfinder,  and  numeric, 
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factors.  We  interpreted  the  six  lower  order-factors  by  ordering  the  factor,  loadings  of  the  tests 
from  high  to  low  and  determining  which  tests  had  the  greatest  causal  influence  on  the  factor 
(Kim  &  Mueller,  1978).  The  first  factor  was  clearly  seen  to  be  a  measure  of  thruput;  the  second, 
response  time;  the  third,  selective  attention;  the  fourth,  tracking;  the  fifth,  pathfinder;  and  the 
sixth,  numerical  operations. 


Table  5.  Correlations  among  First-Order  Factors  (6-Factor  Model) 


Factor 

1 

2 

3 

4 

5  6 

1.  Thruput 

1.0000 

2.  Response  Time 

-0.9010 

1.0000 

3.  Selective  Attention 

0.1624 

-0.0617 

1.0000 

4.  Tracking 

-0.1584 

0.0854 

-0.0881 

1.0000 

5.  Pathfinder 

-0.3400 

0.1553 

-0.0957 

0.2274 

1.0000 

6.  Numerical  Operations 

0.4125 

-0.2415 

0.0577 

-0.2752 

-0.8452  1.0000 

The  correlations  among  the  six  first-order  factors  were  then  used  to  examine  hierarchical 
models.  There  are  two  pairs  of  strongly  correlated  factors:  thruput  and  response  time  and 
pathfinder  and  numerical  operations.  In  keeping  with  previous  findings,  we  tried  a  model  with  a 
single  higher  order  factor  and  foimd  it  to  be  a  poor  fit  to  the  data.  Given  past  experience  (Carretta 
&  Ree,  1996, 1997;  Carretta  et  al,  in  press;  Ree  &  Carretta,  1994;  Stauffer  et  al.,  1996;  Vernon, 
1969),  it  is  surprising  that  the  single  higher  order  factor  model  did  not  provide  a  better  fit.  As 
shown  in  Figure  1,  the  best  fitting  solution  was  for  two  higher  order  factors  that  were  correlated. 
The  correlation  between  these  two  higher  order  factors  was  -.316  indicating  a  shared  source  of 
variance  and  suggests  a  third  level  to  the  hierarchy.  However,  the  existence  of  only  two 
indicators  (2  higher  order  factors)  makes  estimation  of  this  third  level  factor  difficult. 

These  two  higher  order  factors  are  difficult  to  interpret.  The  first  of  these  factors  was 
derived  from  the  thruput,  response  time,  and  shifting  attention  lower  order  factors.  Thruput 
showed  the  highest  loading  on  this  factor.  We  have  interpreted  this  factor  as  a  measure  of 
response  efficiency.  The  other  higher  order  factor  included  tracking,  pathfinder,  and  numeric, 
with  the  latter  two  both  having  the  largest  loadings.  We  had  difficulty  interpreting  this  factor,  but 
have  labeled  it  procedural  knowledge  because  of  the  requirement  to  apply  multiple  rules  to 
complete  the  tests. 

To  understand  the  nature  of  the  CogScreen-AE  factors,  we  conducted  a  qualitative 
content  analysis  of  the  tests  using  Vernon’s  (1969)  3-level  hierarchical  model.  Vernon’s  model 
has  general  cognitive  ability  (g)  at  the  highest  level  and  two  broad  major  group  factors  called 
v:ed  and  k:m  at  the  second  level.  V:ed  is  a  verbal-educational  factor  and  is  typified  by  third-level 
minor  group  factors  such  as  verbal  and  math.  The  k:m  major  group  factor,  called  practical- 
mechanical  is  typified  by  minor  group  factors  such  as  spatial  and  manual.  This  qualitative 
analysis  suggested  that  only  2  of  the  1 1  CogScreen-AE  tests  contribute  to  the  v:ed  factor  while 
the  remaining  9  are  better  categorized  as  measures  of  k:m. 


II 


-.316 


The  higher  order  factor  structure  as  determined  by  our  factor  analysis  does  not  suggest  a 
Vernon-like  structure  (Ree  &  Carretta,  1994;  Carretta  &  Ree,  1997).  The  response  efficiency 
factor  contains  tests  that  can  be  classified  into  both  of  Vernon*s  broad  categories.  The  same  is 
true  of  the  procedural  knowledge  factor.  These  two  higher  order  factors  app>ear  to  include 
response  modality  and  response  process  within  each  factor.  CogScreen-AE  offers  potential  for 
estimating  pilot  candidate  performance  based  on  these  factors. 

Kay  (1995)  offers  interpretations  of  each  test  score  (see  his  Tables  7  through  17).  Because 
factors  aggregate  several  tests,  factors  will  be  more  reliable  than  individual  test  scores.  This 
increased  reliability  raises  the  potential  for  finding  validity  for  the  CogScreen-AE.  Carefully 
conducted  validation  studies  are  required  to  establish  its  validity  and  utility  for  use  with  pilots. 
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